Papers with linear transformation
A La Carte Embedding: Cheap but Effective Induction of Semantic Feature Vectors (P18-1)
Copied to clipboard
| Challenge: | Existing word2vec-based methods for learning rare or unseen words have been criticized for degrading performance in small corpus settings. |
| Approach: | They propose a la carte embedding method that relies on a linear transformation that is efficiently learnable using pretrained word vectors and linear regression. |
| Outcome: | The proposed method is based on a new dataset showing that it can be used when a word is encountered even if only a single usage example is available. |
An Analysis of Euclidean vs. Graph-Based Framing for Bilingual Lexicon Induction from Word Embedding Spaces (2021.findings-emnlp)
Copied to clipboard
Kelly Marchisio, Youngser Park, Ali Saad-Eldin, Anton Alyakin, Kevin Duh, Carey Priebe, Philipp Koehn
| Challenge: | Existing work in bilingual lexicon induction views word embeddings as vectors in Euclidean space. |
| Approach: | They propose to use word embeddings as nodes in a weighted graph to examine a node’s graph neighborhood without assuming a linear transform. |
| Outcome: | The proposed approaches are compared under different data conditions and show that they complement each other when combined. |
Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts (2025.acl-long)
Copied to clipboard
| Challenge: | Prior work has shown that a single LLM’s concept representations can be captured as steering vectors (SVs) prior work has demonstrated that SVs extracted from smaller LLMs can effectively control the behavior of larger LLM. |
| Approach: | They propose a linear transformation method to bridge LLM concept representations using simple linear transformations to enable efficient cross-model transfer and behavioral control via SVs. |
| Outcome: | The proposed method bridges concept representations across different LLMs and enables efficient cross-model transfer and behavioral control via SVs. |
Strong and Simple Baselines for Multimodal Utterance Embeddings (N19-1)
Copied to clipboard
| Challenge: | Human language is a rich multimodal signal consisting of spoken words, facial expressions, body gestures, and vocal intonations. |
| Approach: | They propose two simple but strong baselines to learn embeddings of multimodal utterances by factorizing the utterant into unimodal factors. |
| Outcome: | The proposed models show that they can be derived in closed form while maintaining simplicity and efficiency during learning and inference. |
Unsupervised Multilingual Word Embedding with Limited Resources using Neural Language Models (P19-1)
Copied to clipboard
| Challenge: | Existing methods that map word embeddings into a common space without any parallel data or pre-training have been proposed that are limited in resources and perform poorly under resource-poor conditions. |
| Approach: | They propose a model that maps monolingual word embeddings into a common space without any parallel data and generates multilingual embeddables without any pre-training. |
| Outcome: | The proposed model outperforms existing methods on word alignment tasks on low-resource conditions and with limited resources. |
Information Aggregation for Multi-Head Attention with Routing-by-Agreement (N19-1)
Copied to clipboard
| Challenge: | Existing studies focus on extracting informative or distinct partial-representations from different subspaces, while few studies have paid attention to the aggregation of the extracted partial-Representations. |
| Approach: | They propose to use a routing-by-agreement algorithm to improve multi-head attention by iteratively updating the proportion of how much a part should be assigned to a whole based on agreement between parts and wholes. |
| Outcome: | The proposed algorithm improves the information aggregation for multi-head attention over the standard linear transformation on linguistic probing and machine translation tasks. |
Fully Hyperbolic Neural Networks (2022.acl-long)
Copied to clipboard
| Challenge: | Existing hyperbolic neural networks encode features in the hyperbolical space yet formalize most of their operations in the tangent space. |
| Approach: | They propose a fully hyperbolic framework to build hyperbolical networks based on the Lorentz model by adapting Lorentzer transformations to formalize essential operations of neural networks. |
| Outcome: | The proposed framework has better performance on four NLP tasks compared with existing hyperbolic models . |
A Structural Probe for Finding Syntax in Word Representations (N19-1)
Copied to clipboard
| Challenge: | Existing methods for detecting syntactic knowledge do not test whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Approach: | They propose a structural probe which evaluates whether syntax trees are embedded in a linear transformation of a neural network’s word representation space. |
| Outcome: | The proposed model shows that entire syntax trees are embedded in deep models’ vector geometry. |
Non-Linearity in Mapping Based Cross-Lingual Word Embeddings (2020.lrec-1)
Copied to clipboard
| Challenge: | Existing work on cross-lingual word embeddings rely on linear mappings, but this assumption is not true for all language pairs. |
| Approach: | They propose a non-linear mapping approach which can find non-linesar relationships between languages by kernel Canonical Correlation Analysis. |
| Outcome: | The proposed approach improves on five language pairs on supervised and self-learning scenarios. |
Pyramidal Recurrent Unit for Language Modeling (D18-1)
Copied to clipboard
| Challenge: | Long short term memory units are powerful tools for language modeling, but their performance can be limited by the number of parameters. |
| Approach: | They propose a pyramidal recurrent unit which enables learning representations in high dimensional space with more generalization power and fewer parameters. |
| Outcome: | The proposed model outperforms existing models with different gating mechanisms and transformations on word-level language modeling tasks. |
Cross-Lingual BERT Transformation for Zero-Shot Dependency Parsing (D19-1)
Copied to clipboard
| Challenge: | Existing approaches to learn cross-lingual word embeddings in a contextual space are lacking. |
| Approach: | They propose a method to generate cross-lingual contextualized word embeddings using pre-trained BERT models by learning a linear transformation from contextual word alignments. |
| Outcome: | The proposed approach outperforms state-of-the-art models on zero-shot cross-lingual transfer parsing and is highly competitive with existing models. |
Train Once, Deploy Anywhere: Matryoshka Representation Learning for Multimodal Recommendation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | integrating rich multimodal knowledge into recommender systems remains a challenge . despite performance improvements, different recommendation scenarios often require varying granularities. |
| Approach: | They propose a framework that captures item features at different granularities and learns informative representations for efficient recommendation across multiple dimensions. |
| Outcome: | The proposed framework achieves superior performance over state-of-the-art models on multiple benchmark datasets. |